Nonverbal Behavior Analysis

The last decade marked the emergence of the automated understanding of face-to-face social interaction as a research problem in computing. IM2 was originally focused on meetings (a quintessential form of interaction), and so over the years a body of work directed towards analyzing and inferring a variety of behaviors and interactions resulted from the project. One key aspect of the IM2 work has been the use of nonverbal communication as measurable evidence of social phenomena. The role of nonverbal behavioral cues (gaze, facial expressions, gestures, vocalizations, postures, etc.) as carriers of socially relevant information has been the subject of research in psychology and communication for decades. Furthermore, computing research has developed a large number of approaches aimed at automatic analysis and synthesis of gaze, facial expressions, gestures, and paralanguage. Over 12 years, IM2 enabled both the development of perceptual technologies (computer vision and signal processing) to extract behavioral cues, and their integration to address questions connected to inference of various social variables in increasingly diverse situations (Gatica-Perez, 2009, Vinciarelli et al., 2009b). Table 12.1 shows a timeline of some of the investigated research lines. The initial research that linked audiovisual perception and social behavior in IM2 can be traced back to 2002 with the initial use of the Smart Meeting Room as a sensing platform to study small group interaction (refer to Chap. 1). The original contribution was the dual realization that groups could be studied as units (as opposed to considering individuals as the ultimate analysis target), and that characterizing group behavior computationally could be achieved through the integration of machine perception technologies with findings in the small group research literature. Such a concept took shape through the definition of " meeting actions " , a categorization system that related speaking turns and visual activity, and a framework for recognition using audio and video observations and Hidden Markov Models (McCowan et al., 2005).